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INTRODUCTION 


The  project's  aim  is  to  refine  and  evaluate  practically  a  computer-based  system  that  will 
provide  decision  support  to  the  radiologist  in  interpreting  mammograms  and  generate  a 
standardized  report  of  his/her  findings  for  the  referring  clinician.  For  each  clinical  case,  the  system 
provides  to  the  radiologist  a  checklist  of  the  several  perceptual  features  of  a  mammogram  that  have 
been  determined  to  be  diagnostically  most  relevant.  The  system  elicits  from  the  radiologist  via 
spoken  prompts  a  spoken  numerical  scale  value  for  each  feature  (usually  on  a  10-point  scale).  It 
will  merge  these  values  with  optimal  weights  via  a  statistical  prediction  rule,  to  calculate  a 
probability  of  malignancy  as  an  advisory  for  the  radiologist.  Also,  from  the  pattern  of  feature 
values,  the  planned  system  will  automatically  construct  a  prose  report  of  findings  for  the  referring 
clinician;  spoken  recommendations  and  impressions  will  be  accepted  by  the  system  to  complete  the 
report. 

Earlier  laboratory  experiments  have  shown  that  the  parts  of  the  system  that  aid  mammogram 
reading  and  decision  making  ~  that  is,  checklist  and  statistical  prediction  rule  -  produce  substantial 
increases  in  accuracy  for  the  radiologist  (Getty,  Pickett,  D'Orsi,  and  Swets,  1988);  Swets,  Getty, 
Pickett,  D'Orsi,  Seltzer,  and  McNeil,  1991).  Clinically  oriented  system  evaluations  are  now  to  be 
undertaken  of  refined  versions  of  those  components  and,  as  well,  of  a  refined  and  complete 
version  of  an  automated-reporting  capability.  These  evaluations  will  employ  two  groups  of 
radiologists  and  cases,  one  representative  of  a  community  or  screening  setting  and  the  other,  of  a 
referral  or  diagnostic  setting.  A  significant  difference  between  the  two  settings  is  the  mix  of 
different  types  of  cases.  Another  difference  is  that  the  diagnostic  center  has  the  capability  to  do 
additional  work-up  (e.g.,  magnified  views,  ultrasound)  while  the  patient  is  present  for  the 
mammogram,  which  then  affect  mammogram  interpretation;  in  contrast,  call-backs  are  required  for 
additional  work-up  in  the  screening  setting  and  are  less  frequent.  It  is  possible  that  the  value  of  the 
final,  practical  system  will  be  viewed  differently  in  teaching  hospitals  and  community-oriented 
health  maintenance  organizations  (HMOs). 
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A  principal  refinement  of  the  system  will  be  to  include  observed  changes  in  perceptual 
features  from  previous  to  current  mammograms,  so-called  "interval  change."  These  changes,  as  to 
direction  and  amount,  will  themselves  be  feature  values  for  the  statistical  prediction  rule.  Further, 
various  algorithms  for  merging  feature  values  —  i.e.,  various  algorithms  for  defining  the  statistical 
prediction  rule  —  will  be  tested.  The  algorithm  selected  will  be  designed  to  be  able  to  adapt  over 
time  to  the  mix  of  cases  accumulating  in  the  future  in  each  clinical  setting.  Also,  a  speech- 
reeognition  capability  is  being  incorporated  into  the  system  for  convenient  entry  of  case  data  by  the 
radiologist.  Finally,  software  will  be  developed  to  convert  feature  values  into  a  standardized  prose 
report  of  case  findings;  the  possibility  of  the  system's  effectively  recognizing  concluding  sentences 
dietated  by  the  radiologist  for  the  report  will  be  explored. 

Thirteen  radiologists  will  participate  as  study  readers,  4  in  the  diagnostie  setting  at  the 
Brigham  and  Women's  Hospital  (BWH)  and  8  in  the  screening  setting  at  the  Harvard  Community 
Health  Plan  (HCHP).  Comparison  of  aeeuracies  in  standard  and  aided  eonditions  will  be  made  in 
terms  of  the  relative  operating  characteristie  (ROC)  (Swets,  1979;  Swets  and  Pickett,  1982;  Swets, 
1988;  Dorfman,  Berbaum,  and  Metz,  1992).  Focus  groups  of  clinicians  at  both  sites  will  evaluate 
the  automatieally  eomposed  reports  of  findings  in  comparison  to  the  usual  reports  eomposed  by  the 
radiologist. 

When  utilized  in  practice,  the  system  is  expected  to  promote  quality  assurance  in  several 
respeets  (D’Orsi,  Getty,  Swets,  Piekett,  Seltzer,  and  McNeil,  1992).  Its  accumulating  database  of 
interpretations  and  outcomes  will  provide  a  detailed,  quantitative,  feature-by-feature  basis  for 
evaluating  an  individual's  ability  to  detect  and  assess  perceptual  features  that  distinguish  malignant 
and  benign  conditions,  and  for  examining  differences  among  multiple  opinions.  It  will  provide  a 
basis  for  evaluating  an  individual's  ability  to  set  appropriate  decision  thresholds  for  different  levels 
of  treatment  and  for  adjusting  a  department’s  average  threshold,  e.g.,  as  reflected  in  the 
department's  yield  of  biopsy  reeommendations  (Swets,  1992).  Finally,  the  database  will  allow  the 
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tailoring  of  individualized  tutorials  for  continuing  education  (Greenes,  Swets,  Getty,  and  Pickett, 
1992). 


The  Body  of  the  Report  that  follows  contains  a  section  on  each  of  the  main  technical  tasks  in 
the  project:  1.  Case  Selection;  2.  Checklist  Development;  3.  Automated  Report  Writer;  4.  Data 
Entry  by  Radiologist;  and  5.  Image-Reading  Sessions.  Remaining  sections  are  devoted  to:  6. 
Hardware  and  Software  Acquisition;  7.  Schedule;  and  8.  Personnel. 
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BODY  OF  REPORT 


1 .  Case  Selection 

As  outlined  in  the  original  workplan,  cases  have  been  qualified  and  entered  into  the  study 
from  two  sources  —  Brigham  and  Women's  Hospital  and  Harvard  Community  Health  Plan.  The 
Human  Research  Committees  of  both  institutions  have  approved  the  study  protocol.  Cases  have 
been  enrolled  in  3  categories  -  malignant,  benign,  and  "suspicious  normal".  The  chart  below 
summarizes  eligibility  criteria  and  enrollment  statistics  to  date. 


Method 

of 

Enrolled  at 

Category 

Definition 

Proof 

BWH  HCHP 

Malignant 

All  types  of  breast  cancers 
except  lobular  carcinoma-in-situ 

Pathology 

112 

50 

Benign 

Focal,  nonmalignant  processes 
(i.e.,  benign  tumors) 

Pathology 

156 

50 

Suspicious 

Patient  referred  for  additional 

Clinical/Imaging 

59 

50 

imaging  studies  or  accelerated 

(i.e.,  no  change  in 

follow-up  and  not  returned  to 

lesion  appearance 

routine  screening  pool 

for  24  months  of 
monitoring) 

Patients  enrolled  to  date  fit,  as  expected,  the  demographics  of  the  mammography  referral  (BWH) 
and  screening  (HCHP)  practices  at  our  two  sites.  This  fit  ensures  adequate  enrollment  of  minority 
groups. 


1 . 1  Image  Harvest 

For  each  eligible  ease,  all  available  original  mammographic  and  ultrasound  images  at  the  time 
of  the  "target"  examination  (i.e.,  when  the  suspicious  focus  was  identified)  were  harvested  for  use. 
In  addition,  in  order  to  support  development  and  evaluation  of  "interval  change"  features, 
mammographic  images  from  a  "comparison"  examination  dating  approximately  12  months  (range  6 
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to  1 8  months)  before  the  target  examination  were  also  pulled.  Patient-identifying  information  was 
covered  by  removable  tape  and  a  study  number  assigned  to  each  case  to  ensure  patient 
confidentiality. 

1.2  Image  Quality 

The  quality  of  images  in  each  case  enrolled  at  BWH  was  assessed  by  one  of  the  BWH 
investigators  (T.F.)  who  rated  overall  quality  on  an  ordinal  scale  (1  to  10).  In  addition,  this 
individual  confirmed  that  all  needed  views  were  available  and  confirmed  selection  of  the 
appropriate  "comparison  study." 

1 . 3  Lesion  Location 

In  preparation  for  training  the  statistical  prediction  rule,  another  of  the  BWH  investigators 
(J.E.M.)  reviewed  all  the  selected  images  and  listed  the  coordinates  of  the  most  suspicious 
mammographic  abnormality.  This  step  ensured  that  the  expert  readers  rendered  feature  ratings  on 
the  same  lesion. 

1 .4  Case  Database 

All  available  clinical  mammographic  and  pathology  data  were  summarized  in  an  electronic 
relational  database  (File  Maker  Pro)  to  facilitate  data  extraction  for  the  study. 

1 . 5  Logistics  of  Case  Collection 

As  we  have  found  in  prior  feature-analysis  studies,  identifying  and  qualifying  cases  for  study 
entry  is  a  time-  and  resource-intensive  task.  Our  entry  requirements  were  relatively  stringent  and 
more  demanding  than  in  previous  studies  (including  the  availability  of  "comparison"  films  and,  in 
the  case  of  suspicious  normals,  "follow-up  films").  Therefore,  it  was  necessary  to  peruse  an 
extremely  large  set  of  potential  cases  in  order  to  find  and  qualify  the  requisite  number  for  study 
entry.  In  fact,  because  of  expected  "leakage"  from  our  case  base  (i.e.,  original  films  become 
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unavailable  due  to  clinical  exigencies),  we  qualified  10  to  20%  more  cases  than  the  minimum 
requirement. 


A  further  tax  on  resources  was  our  requirement  that  every  image  on  every  eligible  case  be 
reviewed  by  one  of  the  radiologist  co-investigators  to  ensure  acceptable  image  quality.  The 
retrieval  and  presentation  of  all  these  cases  proved  time  consuming. 

2 .  Completion  of  the  Master  Checklist 

The  master  checklist  is  the  comprehensive  instrument  that  contains  all  of  the  sections  of  items 
our  participating  radiologists  might  follow  as  they  read  cases.  The  particular  sections  followed  will 
depend  on  the  stage  of  our  study  and  the  role  of  the  particular  radiologist  as  either  trainer  or  tester 
of  our  enhanced  system. 

2 . 1  Form  of  the  Master  Checklist 

Work  on  the  master  checklist  began  in  Year  1  and  continued  through  Year  2.  The  finished 
master  checklist  (see  Appendix)  contains  items  for  recording  the  status  of  all  perceptual  features 
considered  of  potential  value  in  accurately  gauging  the  probability  that  a  detected  lesion  is 
malignant,  as  well  as  for  reporting  on  that  probability  and  the  American  College  of  Radiology's 
(ACR)  code  for  recommended  action.  It  also  contains  items  for  reporting  on  the  presence  and 
status  of  clearly  benign  lesions  or  other  conditions  that  would  need  to  be  included  in  a  reasonably 
complete  radiological  report  back  to  the  referring  clinician. 

Because  descriptions  of  clearly  benign  lesions  and  conditions  are  not  needed  to  train  the 
statistical  prediction  rule,  the  BWH  radiologists  will  not  be  required  to  follow  those  sections  of  the 
master  checklist.  When  the  HCHP  radiologists  read  cases  in  our  test  of  the  effectiveness  of  our 
enhanced  system,  they  will  follow  all  sections,  but  those  sections  dealing  with  worrisome  findings 
will  have  been  reduced  to  exclude  those  items  on  the  status  of  a  suspicious  lesion  that  were  found 
in  the  training  process  not  to  be  diagnostically  informative. 
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The  primary  focus  of  work  on  the  master  checklist  in  Year  1  was  to  extend  and  refine  its 
capability  to  support  development  of  the  computer-based  statistical  prediction  rule.  Thus,  we 
concentrated  on  assembling  all  items  of  potential  value  in  classifying  suspicious  lesions  as  benign 
or  malignant.  We  considered  how  best  to  extend  coverage  to  include  interval  change  in  these  same 
features.  We  also  worked  carefully  to  translate  all  of  our  terminology  into  the  recently  approved 
ACR  system. 

2.2  Progress  in  Year  2 

In  Year  2,  we  extended  the  master  checklist  in  several  additional  ways.  We  added  items  for 
reporting  on  various  obviously  benign  lesions  and  other  non-worrisome  findings  that  the  referring 
physician  would  want  or  need  to  hear  about  in  addition  to  information  on  the  absence  or  presence 
of  worrisome  findings.  Also,  because  our  enhanced  system  was  to  be  applied  in  a  real  clinical  and 
screening  context,  it  was  necessary  that  the  questionnaire  include  itenis  for  reporting  on,  and 
merging  into  the  statistical  prediction  rule,  relevant  features  of  the  findings  seen  on  ultrasound 
images. 

The  main  remaining  work  in  Year  2  was  to  review,  test,  and  refine  the  questionnaire  on  the 
basis  of  critiques  from  our  experts  as  they  used  the  questionnaire  to  read  actual  cases.  In  response 
to  the  critiques,  we  have  made  numerous  changes  in  the  checklist  from  where  it  was  at  the  end  of 
Year  1  and  even  as  it  has  evolved  late  into  Year  2.  Several  changes  have  been  made  in  the 
checklist's  overall  organization  to  better  fit  the  sequence  of  considerations  that  experts  prefer  to 
follow,  or  to  put  particular  items  into  sections  that  contain  other  items  with  which  they  are  usually 
considered.  For  example,  it  seemed  logical  enough  for  the  checklist  designers  to  put  the  item  on 
"Presence  of  Scattered  Calcifications"  into  a  section  dealing  with  "General  Breast  Background," 
but  the  experts  preferred  that  this  item  be  moved  into  a  restructured  and  more  general  version  of  a 
section  we  had  narrowly  devoted  to  "Calcification  Clusters." 
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In  addition,  many  items  that  we  had  inserted  or  planned  to  insert  in  the  checklist  were,  in 
view  of  the  burgeoning  size  of  the  checklist,  dropped  as  being  too  specific  or  too  overlapping  to 
justify.  For  example,  we  originally  aimed  to  have  the  ratings  of  "Percent  Glandular  Tissue"  made 
separately  for  each  breast,  but  soon  opted  for  the  more  practical  approach  of  getting  a  summary 
judgment  for  the  two  breasts.  Similarly,  we  originally  envisioned  getting  interval-change 
assessments  on  each  of  the  key  diagnostic  items  for  every  finding,  but  that  soon  appeared  to  be 
impractical.  In  the  final  version  of  the  checklist,  the  radiologist  reports  for  each  worrisome  finding 
whether  it  shows  any  significant  change  from  the  prior  study.  Only,  if  it  is  so  judged,  is  the 
radiologist  required  to  go  back  and  assess  all  the  key  items  on  the  prior  study. 

Finally  -  working  especially  with  the  co-investigator  who  co-developed  the  ACR  lexicon 
(C.J.D.)  —  we  made  several  changes  aimed  at  increasing  the  clarity  of  the  checklist  items  and  the 
precision  of  the  radiologists'  reports.  For  example,  in  considering  the  nature  of  calcifications,  the 
ACR  system  requires  only  that  the  radiologist  check  whether  a  particular  type  of  caleification 
element  is  present.  Initially,  we  followed  that  approach.  But  the  critiques  revealed  that  we  could 
get  more  precise  information.  Now  the  checklist  items  ask  for  the  degree  to  which  the  radiologist 
is  confident  that  each  type  is  present,  and  provide  for  the  rating  of  confidence  to  be  made 
quantitatively  on  a  precisely  labeled  scale.  Not  only  does  this  provide  for  obtaining  more  precise 
information  and  ultimately  more  accurate  diagnoses  in  our  studies,  but  it  provides  suggestions  for 
ways  that  the  ACR  system  can  ultimately  be  improved. 

We  believe  that  we  have  achieved  in  this  final  version  of  the  checklist  a  practical  instrument 
of  high  potential  informativeness  for  the  present  studies  as  well  as  an  exemplary  and  promising 
system  for  field  application. 

3 .  Automated  Report  Writer 

The  automated  report  writer  will  render  the  primarily  numerical  information  from  the 
questionnaire  in  ordinary  English.  This  translation  provides  a  natural  summary  of  the  information 


12 


without  burdening  the  radiologist  with  the  task  of  producing  a  report  and  it  can  serve  to  standardize 
radiology  reports  so  that  they  contain  the  same  information  and  describe  similar  judgments  from 
different  radiologists  in  similar  terms. 

We  had  two  major  goals  for  Year  2  of  the  project: 

1 .  Collect  radiology  reports  and  other  data  to  determine  how  a  radiologist  would 
express  the  information  in  the  questionnaire. 

2 .  Design  an  architecture  for  the  report  writing  system. 

In  addition  to  meeting  both  of  those  goals,  there  was  a  third,  unintended  benefit  from  the 
work,  which  was  to  contribute  to  refining  the  questionnaire.  Through  the  text  analysis  of  actual 
radiology  reports  we  were  able  to  notice  places  where  the  questionnaire  was  slightly  vague, 
imprecise,  or  missing  information.  This  led  to  changes  in  wording  in  several  questions  and  the 
addition  of  a  few  new  questions. 

In  this  section,  we  describe  the  accomplishments  in  these  two  areas  and  outline  our  plans  for 
the  next  year. 

3 . 1  Data  Analysis 

We  examined  two  kinds  of  data  in  this  phase  of  the  work.  The  first  is  targeted  data,  that  is, 
reports  that  directly  express  the  information  from  the  questionnaire.  The  second  is  general  data 
collected  from  radiology  reports. 

3.1.1  T  argeted  Reports 

The  first  set  of  targeted  data  was  obtained  by  having  a  radiologist  fill  out  a  questionnaire  and 
then  dictate  a  report  of  the  information  (for  ten  cases).  This  kind  of  data  is  the  most  useful  because 
it  provides  direct  information  about  how  the  items  in  the  questionnaire  are  expressed.  We  are  in 
the  process  of  collecting  a  second  set  of  targeted  data  that  will  cover  the  same  cases,  but  will  be 
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produced  by  a  different  radiologist,  which  will  provide  some  information  on  variation  in  how  the 
questions  can  be  expressed.  We  will  also  be  collecting  a  much  larger  set  of  reports  as  part  of  the 
data  collection  portion  of  the  project. 

Analysis  of  these  collected  reports  for  the  text  generator  led  us  to  note  some  improvements 
that  could  be  made  in  the  questionnaire,  as  shown  in  the  following  two  examples.  In  case  181, 
we  found  that  although  the  radiologist  described  the  mass  as  “circumscribed,"  there  was  no  direct 
item  in  the  questionnaire  that  included  this  information.  This  was  one  factor  that  led  to  the  addition 
of  a  new  question.  The  wording  of  the  calcification  description  in  case  184  as  “fine  linear”  led  to  a 
change  in  the  wording  on  the  questionnaire.  It  originally  asked  about  “fine/linear  calcifications”, 
which  implied  it  could  be  “fine  or  linear”  and  was  changed  to  “fine  linear”  to  be  in  keeping  with  the 
report. 

Case  181:  Bilateral  mammograms.  Dense  heterogeneous  glandular  tissue  is  seen.  There  is 
a  circumscribed  lobular  mass  situated  at  the  approximately  9  o’clock  posterior  position  of  the 
right  breast.  Remainder  of  the  exam  is  within  normal  limits. 

Impression:  Mass  as  described  above.  Ultrasound  should  be  done  for  further  categorization. 
Please  code  as  category  zero  (0). 

Case  184:  Bilateral  mammograms.  Dense  heterogeneous  glandular  tissue  was  seen.  There 
is  a  linear  collection  of  fine  linear  pleomorphic  calcifications  in  the  anterior  subareolar  area  on 
the  left  which  is  markedly  changed  from  the  prior  exam  of  one  year  previous. 

Impression:  suspicious  calcifications  as  described  above.  Localization  biopsy 
recommended.  Please  code  as  category  four  (4). 

3.1.2  General  data 

The  general  data  were  provided  from  Brigham  &  Women’s  Hospital.  We  have  845  reports 
(about  100,000  words)  of  these  data  and  are  using  them  to  look  at  variations  in  how  certain  words 
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and  phrases  are  used.  We  have  a  tool  that  cdlows  us  to  search  for  all  sentences  that  contain  a 
certain  word  or  phrase.  For  example,  we  found  89  sentences  that  have  the  word  “circumscribed” 
in  them.  Almost  all  (76)  say  “well  circumscribed,"  However  there  are  some  variations,  such  as 
“fairly,"  “smoothly,"  and  “new”  as  in  examples  3,  4,  and  5  below.  Also  note  that  what  is 
circumscribed  is  almost  always  described  as  a  “nodule."  However,  again,  there  are  some 
exceptions,  as  shown  in  6  and  7,  which  are  describing  “masses”  and  “lesions." 

1 .  The  previously  noted  well  circumscribed  left  upper  outer  quadrant  nodule  has 
remained  unchanged  from  the  examination  of  12-8-88  and  is  consistent  with  a  lymph 
node. 

2.  Breast  ultrasound  demonstrates  this  to  be  a  relatively  well  circumscribed  hypoechoic 
nodular  density  with  some  through  transmission  of  sound. 

3 .  Very  posteriorly  in  the  left  slightly  inner  upper  breast  a  lobulated  ^ir/y  circumscribed 
nodule  is  again  noted. 

4.  A  new  smoothly  circumscribed  nodule  is  seen  within  the  right  upper  inner  breast 
which  was  not  present  on  the  previous  exam. 

5 .  New  circumscribed  nodule  left  slightly  upper  outer  breast  which  is  shown  to  be  cystic 
on  breast  ultrasound. 

6 .  Numerous  well-circumscribed  partially  obscured  masses  are  present  bilaterally. 

7 .  Multiple  well  circumscribed  lesions  are  compatible  with  cysts. 

These  data  can  be  used  to  augment  the  individually  collected  reports  in  deciding  how  the 
report  writer  should  express  certain  kinds  of  information.  We  can  also  look  at  how  information 
can  be  combined,  such  as  folding  locations  or  comparisons  with  previous  studies  into  a  single 
sentence. 
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Although  data  analysis  is  an  important  part  of  determining  how  the  primarily  numerical 
information  in  the  questionnaire  can  be  expressed  in  English,  we  must  also  incorporate  the 
guidelines  provided  by  the  American  College  of  Radiology,  because  our  goal  is  not  just  to  make 
the  communication  natural,  but  to  also  make  it  consistent.  The  “Breast  Imaging  Reporting  and 
Data  System”  (BI-RADS™)  document  provides  both  a  lexicon  and  the  recommended  report 
structure  that  we  will  be  incorporating  into  the  reporting  system. 

3.2  Architecture 

The  architecture  of  the  system  should  be  modular,  so  that  we  can  begin  with  a  simple 
prototype  and  then  be  able  to  be  incrementally  extend  it  as  more  sophistication  is  needed.  It  must 
also  be  easy  to  change  the  way  the  system  expresses  information,  so  that  in  the  testing  phase  we 
can  have  the  radiologists  use  and  critique  the  system,  and  then  make  adjustments. 

3.2.1  Report  Grammar 

Our  approach  is  to  use  a  “grammar”  to  provide  the  top  level  organization  of  the  report,  as 
shown  by  the  grammar  rule  below  which  is  based  on  the  BI-RADS.  A  grammar  can  be  thought  of 
a  template  that  defines  what  information  should  be  included  and  how  it  should  be  organized. 
"Non-terminals"  to  the  left  of  the  arrow  are  expanded  to  what  is  on  the  right  of  the  arrow. 

REPORT  =»  COMPOSITION  FINDINGS  ASSESSMENT 

Each  of  the  parts  (e.g.  COMPOSITION  and  FINDINGS)  is  then  expanded  by  the  grammar 
to  include  its  parts,  which  can  be  words  or  another  non-terminal  in  the  grammar,  which  also  must 
be  expanded.  When  there  are  multiple  choices  of  ways  to  expand  a  non-terminal,  the  choice  is 
conditioned  on  the  numeric  values  in  the  questionnaire.  We  have  included  a  preliminary  version  of 
the  questionnaire  in  the  appendix  of  this  document.  Using  that  grammar,  we  could  produce  a 
paragraph  such  as  the  one  below,  which  is  based  on  Case  181. 
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The  breast  is  heterogeneously  dense.  There  is  a  10  mm  circumscribed  lobular  mass  at  the  9 

o’clock  position  of  the  right  breast.  This  is  a  new  finding. 

This  first  prototype  will  allow  us  to  quickly  produce  simple  reports  so  we  can  get  early 
feedback.  As  we  progress,  we  will  incorporate  a  text  generation  system  that  can  integrate 
information  fluently  into  sentences,  rather  than  just  print  out  each  sentence  or  part  separately,  as  is 
done  by  this  grammar. 

3.2.2  Specialists 

Another  important  part  of  our  approach  is  isolate  kinds  of  information  that  can  occur  in  many 
different  kinds  of  reports  or  multiple  times  in  the  same  report  and  use  text  generation  “specialists” 
to  produce  the  descriptions.  A  good  example  of  this  is  location.  While  there  are  many  different 
kinds  of  findings,  such  as  masses,  lesions,  calcifications,  and  architectural  distortions,  there  are 
relatively  few  ways  of  describing  the  locations  of  these  findings.  The  grammar  should  have  one 
“location  specialist,"  that  can  handle  the  various  ways  location  is  expressed,  whether  focal,  such  as 
o’clock  position,  or  regional. 

Determining  which  specialists  should  be  created  and  where  they  are  used  is  a  complex 
problem,  given  the  many  different  parts  of  the  questionnaire,  each  with  many  questions.  A 
radiologist  filling  in  information  rarely  uses  or  even  sees  all  of  these  parts,  since  only  those  for  the 
findings  he  or  she  specifies  are  shown.  However,  the  text  generator  must  be  ready  to  produce 
descriptions  for  all  of  them  in  case  they  are  needed.  In  order  to  determine  which  specialists  to 
create  and  where  they  should  be  used,  we  have  put  the  many  sections  of  the  questionnaire  into  a 
table  with  all  of  the  questions  asked.  We  can  then  see  patterns  of  questions  that  might  be  best 
handled  with  specialist  and  places  where  a  part  needs  to  be  handled.  A  portion  of  the  table  is 
included  in  the  appendix. 
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3.3  Plans 


The  major  tasks  we  will  be  undertaking  in  the  next  year  of  the  project  are: 

1 .  Continue  collection  and  analysis  of  report  data. 

3 .  Design  and  implement  the  interface  between  the  overall  system  and  the  report  writer. 

4.  Implement  a  prototype  version  of  the  grammar,  focusing  on  the  organization  of 
information  and  using  canned  text  for  the  actual  output. 

5 .  Begin  integrating  the  system  with  more  sophisticated  text  generation  techniques. 

4 .  Data  Entry  by  Radiologist 

Extensive  software  was  written  to  collect  the  test  radiologists'  responses  to  the  checklist 
questionnaire  described  in  Section  2  above.  Data  entry  by  the  radiologist  is  in  speech  form,  in 
response  to  auditory  presentation  of  the  checklist,  with  intelligent  interactive  passage  through 
relevant  portions  of  the  checklist. 

4.1  Hardware  and  Software 

We  adopted  as  hardware/software  for  speech  recognition  the  Phonetic  Engine  500  system 
from  Speech  Systems  Inc.  The  software  had  bugs  that  we  identified  and  communicated  to  the 
developer  and  we  tested  the  new  software  components  supplied  to  us  to  upgrade  the  system.  The 
need  for  additional  computer  memory  was  identified  and  satisfied,  with  a  resulting  improvement  in 
recognition  performance.  We  adopted  a  push-to-talk  method  as  a  replacement  for  the  system's 
voice-activation  algorithm,  with  a  resulting  large  improvement  in  accuracy  and  speed.  We  began 
testing  the  beta  version  of  the  package  designed  for  Windows  95  in  anticipation  of  moving  from 
Windows  3.1. 
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4.2  Specifics  of  Speech  Interaction 


In  implementing  the  checklist  for  spoken  data  entry,  we  solved  the  problems  created  by 
speech  interaction  that  are  not  present  with  keyboard  data  entry  and  visual  feedback.  Specifically, 
questions  and  responses  in  written  form  were  modified  and  abbreviated  for  the  auditory  format; 
decisions  were  made  about  how  to  replace  keyboard  input  with  speech  input  and  how  to  provide 
auditory  feedback  in  addition  to  visual  feedback;  and  a  process  was  implemented  to  permit  the 
radiologist  to  change  responses  orally. 

4 . 3  Database  of  Perceptual  Features 

A  database  of  the  perceptual  features  incorporated  in  the  checklist  was  constructed  to  drive 
the  data  collection  process.  Written  in  EXCEL,  the  database  contains  information  on  each  of  the 
360  questions  that  may  be  asked.  It  was  written  to  permit  changes  to  the  checklist's  text  and  its 
flow  without  requiring  changes  in  software.  It  provides  the  question  identifier;  the  text  for  screen 
presentation;  a  pointer  to  the  digital  audio  file  that  contains  the  auditory  prompt;  a  pointer  to  the  file 
that  contains  the  possible  responses  in  auditory  form;  any  question  the  present  question  may 
depend  on;  the  next  question  if  sequential  or  which  of  two  questions  to  go  to  depending  on  the 
answer  to  the  present  question;  and  the  name  of  the  syntax  file  for  the  question. 

4.4  Syntaxes 

We  developed  a  syntax  for  each  of  the  sets  of  responses  to  the  various  questions,  preferring 
to  develop  separate  syntaxes  for  each  question,  rather  than  one  general  syntax,  to  assure  high 
recognition  accuracy. 

4.5  Speech  Recording 

All  the  speech  heard  by  the  radiologist  was  recorded  and  digitized:  including  prompts  to 
questions,  feedback  in  the  form  of  speaking  what  the  recognizer  recognized,  and  possible 
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responses  to  the  questions.  An  index  file  was  created  to  link  text  words  used  in  the  interactive 
process  to  the  digital  file  containing  the  recorded  speech. 

4.6  Software  for  Experimental  Control  and  Data  Analysis 

A  program  was  written,  using  Visual  Basic,  to  control  the  reading  session.  The  program 
identifies  the  reader  and  the  session  and  uses  this  information  to  open  a  data  file  specifying  the 
cases  to  be  read  by  the  mammographer  and  their  order.  For  each  successive  case,  the  program 
presents  the  reader  with  the  case  number  and  the  patient's  date-of-birth.  It  then  guides  the  reader 
through  the  reading  of  the  case,  first  identifying  a  lesion  and  then  rating  features  associated  with 
that  lesion  type,  prompting  the  reader  with  a  spoken  cue  for  each  feature  to  be  rated.  Feature 
values  spoken  by  the  reader  are  collected  by  the  program  and  stored  in  a  data  file. 

Another  program  was  developed  to  read  the  data  created  in  the  interactive  process,  organize  it 
into  a  full  table  format  that  represents  all  of  the  questions,  and  then  write  out  the  data  in  this  format 
for  analysis. 

Constructing  this  collection  of  computer  programs  provided  a  setting  in  which  we  reviewed 
the  checklist  for  consistency  and  logical  flow. 

5.  Image-Reading  Sessions 

We  scheduled  image-reading  sessions  for  the  5  participating  radiologists  at  BWH  in  August, 
but  postponed  them  when  the  case-selection  and  qualification  processes  were  found  to  require 
more  time  and  effort  than  planned.  As  a  consequence,  we  did  not  collect  the  feature-rating  data  that 
serve  to  train  the  statistical  prediction  rule.  We  plan  now  to  eonduct  the  BWH  image-reading 
sessions  in  the  first  quarter  of  Year  3,  and  to  develop  the  merging  algorithms  of  the  statistical 
prediction  rule  and  conduct  HCHP  baseline  image-reading  sessions  in  the  second  quarter. 
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6 .  Hardware  and  Software  Acquisition 


To  facilitate  easy  viewing  of  current  and  comparison  study  mammograms,  two 
mammographic  viewboxes  were  acquired.  These  MAX  ANT  Mammolume  MM  0211  systems 
have  features  that  are  well-matched  to  the  study  requirements,  including:  two-tiered  design 
(allowing  film  viewing  in  a  4-over-4  format);  interior  shutters  (allowing  viewing  of  8x10  and 
10x12  in  film  sizes  simultaneously  and  intermittently  without  surrounding  glare);  high-intensity 
illumination  and  portability. 

While  no  new  software  was  acquired  in  Year  2,  periodic  upgrades  to  statistical  and  voice- 
recognition  software  were  obtained. 

7 .  Schedule  and  Level  of  Effort 

The  project  ran  at  a  level  of  effort  and  expenditure  through  Years  1  and  2  of  about  90%  of 
budget.  Although  parts  of  the  project  were  not  completed  on  the  original  schedule,  we  expect  that 
they  will  be  completed  in  the  first  few  months  of  the  next  project  year  and  that  the  end  of  Year  3 
will  find  us  on,  or  possibly  ahead  of,  schedule. 

8 .  Personnel 

All  staff  and  consultants  continue  with  the  project.  Reduced  time  of  research  associate  Lisa 
Hermann,  Pharm.  D.,  has  been  balanced  by  the  addition  of  Mr.  Edward  Chao,  who  joined  the 
Brigham  and  Women's  Hospital  staff  after  a  1994  B.S.  from  UCLA  and  two  years  of  hospital 
experience. 


21 


CONCLUSIONS 


Our  major  accomplishments  in  Year  2  include,  first,  dealing  successfully  with  complexities 
and  problems  that  arose  in  the  definition  of  an  appropriate  set  of  cases  for  training  the  statistical- 
prediction  rule  at  the  BWH,  and  with  the  effortful  requirements  of  selecting  and  qualifying 
individual  cases  to  meet  the  definition,  as  well  as  identifying  the  location  of  the  principal  lesion  in 
each. 


Second,  we  developed  a  much-improved  version  of  the  checklist  of  perceptual  features:  we 
added  items  on  non-worrisome  findings  and  on  ultrasound,  tested  the  checklist  on  our  experts, 
restructured  it  to  conform  to  the  order  preferred  by  our  experts,  pruned  it  to  consider  length  as  well 
as  inclusiveness,  and  increased  its  clarity. 

Third,  the  automated  report  writer  was  advanced  appreciably.  Radiology  reports  were 
analyzed  in  relation  to  checklist  items  -  both  a  small  sample  of  reports  written  along  with  a 
completed  checklist  on  a  case-by-case  basis,  and  a  large  sample  of  reports  written  previously  to 
look  for  natural  semantic  variation.  The  architecture  of  the  report  writer  was  designed  and 
developed. 

Fourth,  extensive  programming  was  completed  to  handle  computer  entry  of  spoken  data  by 
the  radiologist,  as  the  checklist  is  followed,  and  the  verbal  reply  of  the  computer.  A  speech- 
recognition  software  package  was  debugged.  Software  was  written  for  a  database  of  perceptual 
features,  for  syntaxes  of  responses,  for  control  of  an  image-reading  session,  and  for  data  analysis. 

Fifth,  the  convenience  of  image  reading  (by  a  dozen  radiologists  at  two  sites)  was  enhanced 
by  our  acquisition  of  two  portable  mammographic  viewboxes,  which  allow  intermittent  reading  of 
different-sized  films  without  surrounding  glare. 
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Our  one  regret  is  that  the  time  and  effort  expended  well  on  these  accomplishments  were  so 
large  as  to  prevent  us  from  accomplishing  also  the  formal  image-reading  sessions  that  were 
planned  for  Year  2.  These  sessions  are  now  planned  for  the  first  half  of  Year  3. 

Year  3  will  contain  the  image-reading  sessions,  one  of  which  will  provide  data  to  train  a 
statistical-prediction  rule  and  define  the  final,  reduced  checklist  of  image  features,  and  others  of 
which  will  provide  baseline  data  and  reading  data  enhanced  by  the  checklist  and  prediction  mle. 

Year  3's  other  major  task  is  completion  and  preliminary  testing  of  the  automated  report 
writer. 

Year  4  is  planned  to  include  evaluation  of  the  report  writer  with  focus  groups  of  clinicians 
who  typically  receive  prose  reports  of  mammographic  findings.  Year  4  will  also  see  final  data 
analysis  and  preparation  of  the  final  report. 
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APPENDIX  A 


Response  Form 
X-Ray  Mammography 


Reader  No.. 
Case  No._ 


Overview  of  Breast  Images 

Image  Quality 

0123456789  10 

poor  excellent 

Percentage  of  Tissue  that  is  Glandular 

_ % 

Density  of  glandular  tissue 

0  1  2  3  4  5  6  7  8  9  10 

slightly  denser  much  denser 

than  fatty  tissue  than  fatty  tissue 

Findings? 


OVOl 


OV02 


OV03 


o 

None 

o 

One  or  more 

If  one  or  more  findings,  list 

in  order  of  priority  (from  most  suspicious  to  most  benign)  all  the  findings  that 

are  to  be  reported: 

Mass 

Secondary  Signs 

MM  =  Mass 

AA  =  Axillary  Adenopathy 

1. 

Soecial  Case  Densities 

NR  =  Nipple  Retraction 

SR  =  Skin  Retraction 

2. 

AT  =  Asymmetric  Breast  Tissue 

ST  =  Skin  Thickening 

3. 

TD  =  Tubular  Density  (Solitary  Dilated  Duct) 

TT  =  Trabecular  Thickening 

Architectural  Distortion 

4. _ 

AD  =  Architectural  Distortion 

5. 

Calcifications  (Not-Definitelv-Benign) 

6.  _ 

NC  =  Not-Definitely-Benign  Calcifications 

7. _ 

Calcifications  (Definitely  Benign) 

SC  =  Skin  Calcifications 

8. 

VC  =  Vascular  Calcifications 

CC  =  Coarse  (“Popcorn-Like”)  Calcifications 

RL  =  Large  Rod-Like  Calcifications 

LC  =  Spherical  Lucent-Centered  Calcifications 

EC  =  Eggshell  (Rim)  Calcifications 

MC  =  Milk  of  Calcium 

9. 

10. 

UC  =  Suture  Calcifications 

DC  =  Dystrophic  Calcifications 
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Reader  No.. 
Case  No._ 
Finding  No._ 

Mass  CMM) 


•  Confidence  regarding  the  presence  of  a  mass 


0  1 
definitely  NOT 
present 


23456789  10  MMOl 

definitely  present 


•Distribution 

O  single  mass 

O  multiple  similar  masses  MM02 

•Locus  -  breast 

O  left  breast  O  right  breast  MM02A 


•  Locus  -  within  breast  (If  single  mass): 


O  O’clock  position 


O  anterior 
O  middle 
O  posterior 


MM03A 

MM03B 


O  central 

O  anterior  (subareolar) 
O  middle 
O  posterior 

O  axillary  tail 


•Density  of  mass  relative  to  surrounding  glandular  tissue 


0 

mass 
density 
much  lower 


2  3  4  5  6  7  8 

isodense 


9  10 

mass 
density 
much  higher 


MM04 


27 


Mass  (MM)  -  cont, 


•Confidence  about  the  presence  of  fat  within  the  mass 


0123456789  10 

definitely  NONE  definitely  some 

present  present 


MM05 


•  Size  of  mass 


Smallest  diameter  in  CC  view  _ 

mm 

MM06 

Smallest  diameter  in  lateral  view  _ 

mm 

MM07 

Largest  diameter  in  CC  view  _ 

mm 

MM08 

Largest  diameter  in  lateral  view  _ 

mm 

MM09 

•  Shape  of  mass 


0123456789  10 

round/oval  lobular  irregular 


MMIO 


•Confidence  that  at  least  a  small  portion  of  the  margin  is  indistinct  due  to  tissue  invasion 


0  1  2  3  4  5 

definitely  NONE 
of  margin  indistinct  due 
to  tissue  invasion 


6  7  8  9  10  MMll 

definitely  some 
of  margin  indistinct  due 
to  tissue  invasion 


•Confidence  that  at  least  a  small  portion  of  the  margin  is  spiculated 


0  1 
definitely  NOT 
spiculated 


10 

definitely 

spiculated 


MM12 


•  Percentage  of  the  margin  that  is  . . .  (total  should  add  to  up  100%) 


A.  clearly 
circumscribed 


B.  obscured  by 
%  glandular  tissue 


C.  indistinct  due  to 
tissue  invasion 


_%  spiculated 


MM13A 
MM13B 
%  MM13C 


•Degree  of  microlobulation 

0123456789  10  MM14 

none  extensive 
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Mass  fMM)  -  coni 


•  Confidence  that  the  mass  is  a  skin  lesion 


0  1 

2 

3 

4 

5 

6 

7 

8 

9  10 

MM15 

definitely  NOT 

definitely 

a  skin  lesion 

a  skin  lesion 

•Confidence  that  the  mass  is  an  intramammary  node 


0  1  2 

3 

4 

5 

6 

7 

8 

9  10 

MM16 

definitely  NOT 

definitely 

an  intramammary  node 

an  intramammary  node 

•Confidence  regarding  the  presence  of  related  architectural  distortion 
0  1  2  3  4  5  6  7  8  9  10  MM17 

definitely  NOT  definitely 

present  present 


•Confidence  regarding  the  presence  of  worrisome  calcifications  within  the  mass 

0123456789  10  MM18 

definitely  NOT  definitely 

present  present 


•Confidence  regarding  the  presence  of  benign  calcifications  within  the  mass 


0  1 
definitely  NOT 
present 


2  3  4  5  6  7 


8 


9  10 

definitely 

present 


MM19 
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•Is  there  an  ultrasound? 


O  Yes 
O  No 


)uestions  relating  to  Ultrasound 


•Appearance  of  the  mass  wall  (ultrasound) 
O  well  circumscribed 

O  indistinct 

O  irregular 


•Contents  of  the  mass  (ultrasound) 
O  solid 

O  indeterminate 

O  cystic 


•Response  of  the  posterior  wall  of  the  mass  (ultrasound) 
O  enhancement 

O  iso-echoic 

O  shadowing 


•  Shape  of  the  mass  (ultrasound) 
O  round 

O  ellipsoid 

O  irregular 


MMUL 


MMULl 


MMUL2 


MMUL3 


MMUL4 


Mass  (MM)  -  cont. 


Relationship  to  Prior  Study 

•This  mass  finding  is: 

O  new 

O  significantly  changed 
O  unchanged  or  not  significantly  changed 


MM20 

MM21 
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Asymmetric  Breast  Tissue  T  A 


Reader  No._ 
Case  No.. 
Finding  No._ 


•Confidence  regarding  presence  of  asymmetric  breast  tissue 


012345678 
definitely  NOT 
present 


9  10 

definitely 

present 


ATOl 


•Distribution  of  asymmetric  breast  tissue 


O  focal 

AT02 

O  global 

•Locus  -  breast 

O  left  breast 

O  right  breast 

AT03 

•  Locus  -  within  breast 

(If  focal  distribution): 

AT04A 

AT04B 

O  O’clock  position  _ 

O  anterior 
O  middle 
O  posterior 


O  central 

O  anterior  (subareolar) 
O  middle 
O  posterior 

O  axillary  tail 
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Asymmetric  Breast  Tissue  TAT)  -  cont. 

•  Size  of  distribution  of  asymmetric  breast  tissue  (If  focal  distribution) 


Largest  diameter  in  CC  view  _ 

mm 

AT05 

Largest  diameter  in  lateral  view 

mm 

AT06 

•Confidence  regarding  the  presence  of  worrisome  calcifications  within  the  asymmetric 
breast  tissue 


0  1 
definitely  NOT 
present 


2  3  4  5  6  7 


8 


9  10 

definitely 

present 


AT07 


•  Confidence  regarding  the  presence  of  benign  calcifications  within  the  asymmetric 
breast  tissue 


0  1 
definitely  NOT 
present 


2  3  4  5  6  7  8 


9  10 

definitely 

present 


AT08 


•  Confidence  regarding  presence  of  related  architectural  distortion 


0  1 
definitely  NOT 
present 


2  3  4  5  6  7 


8 


9  10 

definitely 

present 


AT09 


Relationship  to  Prior  Study 


•This  asymmetric  breast  tissue  finding  is:  ATIO 

ATll 

O  new 

O  significantly  changed 
O  unchanged  or  not  significantly  changed 
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Reader  No.^ 
Case  No.. 
Finding  No._ 

Tubular  Density  (Solitary  Dilated  DucO 


•Confidence  regarding  the  presence  of  a  tubular  density  (solitary  dilated  duct) 


0  1 
definitely  NOT 
present 


2  3  4  5  6  7 


8 


9  10 

definitely 

present 


•Locus  -  breast 

O  left  breast  O  right  breast 

•Locus  -  within  breast 

O  O’clock  position _  O  anterior 

O  middle 
O  posterior 


TDOl 


TD02 


TD03A 

TD03B 


O  central  O  anterior  (subareolar) 
O  middle 
O  posterior 

O  axillary  tail 

Relationship  to  Prior  Study 


•This  finding  is:  TD04 

TD05 

O  new 

O  significantly  changed 
O  unchanged  or  not  significantly  changed 
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Architectural  Distortion  (AD) 


Reader  No._ 
Case  No.. 
Finding  No._ 


•Confidence  regarding  presence  of  architectural  distortion  in  a  least  one  view 


0  1  2  3  4  5  6  7 

definitely  NONE 
present 


8  9  10 

definitely  some 
present 


ADOl 


•  Locus  -  breast 


O  left  breast  O  right  breast 


AD02 


•Visible  in  both  views  or  only  one? 

O  both 

O  one:  Oblique 
O  one:  CC 

ADOS 


•Locus  -  within  breast 


If  both  views: 


If  Oblique: 


If  CC: 


O  O’clock  position  _ 

O  anterior 
O  middle 
O  posterior 

O  central 

O  anterior  (subareolar) 

O  middle 
O  posterior 
O  axillary  tail 

O  superior 
O  middle 
O  inferior 
O  axillary  tail 

O  medial 
O  middle 
O  lateral 


AD04A 

AD04B 


AD05A 


AD05B 
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Architectural  Distortion  (AD^  -  cont. 


•Confidence  that  the  architectural  distortion  is  related  to  prior  surgery 


0  1  2  3  4  5 

definitely  NOT 
related  to  prior  surgery 


6  7  8  9  10 

definitely 

related  to  prior  surgery 


•Confidence  regarding  the  presence  of  related  worrisome  calcifications 

0123456789  10 

definitely  NOT  definitely 

present  present 


•Confidence  regarding  the  presence  of  related  benign  calcifications 


012345678 
definitely  NOT 
present 


9  10 

definitely 

present 


•Confidence  regarding  the  presence  of  a  related  mass 


0  1 
definitely  NOT 
present 


2 


4  5  6  7  8 


9  10 

definitely 

present 


•Confidence  regarding  the  presence  of  related  asymmetric  breast  tissue 


0  1 
definitely  NOT 
present 


2  3  4  5  6  7  8 


9  10 

definitely 

present 


Relationship  to  Prior  Study 


•This  architectural  distortion  finding  is: 

O  new 

O  significantly  changed 
O  unchanged  or  not  significantly  changed 


AD06 


AD07 


AD08 


AD09 


ADIO 


ADll 

AD12 
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Reader  No. _ 

Case  No. _ 

Finding  No. _ 

Calcifications  (Not-Definitelv-Benign) 

•Confidence  regarding  the  presence  of  not-definitely-benign  calcifications 
0  1  2  3  4  5  6  7  8  9  10  NCOl 

definitely  NOT  definitely 

present  present 

•Distribution  of  calcifications  (choose  one): 

o  single  focal  distribution  (clustered,  linear,  segmental) 

O  multiple  similar  focal  distributions 
O  regional  distribution 

O  diffuse  NC02 


•Locus  -  breast 

O  left  breast  O  right  breast 


NCOS 


•Locus  -  within  breast  (If  single  focal  or  regional  distribution): 


If  single  focal 


If  regional 


NC04A 

NC04B 

NC04C 

NC04D 


O  O’clock  position  _ 

O  anterior 
O  middle 
O  posterior 

O  central 

O  anterior  (subareolar) 

O  middle 
O  posterior 
O  axillary  tail 


(check  as  many  as  apply) 

Q  superior  medial 
n  superior  lateral 
n  inferior  medial 
D  inferior  lateral 
D  central 

D  anterior  (subareolar) 
D  middle 
n  posterior 
n  axillary  tail 
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Calcifications  (Not-Definiteiv-Benign)  (NC)-cont. 

Elemental  Characteristics 

•Size  of  largest  individual  element  (best  visual  estimate)  NCOS 

less  than  0.5  mm  0.5  mm  to  1.0  mm  greater  than  1.0  mm 

•Variability  of  size  of  elements 

0123456789  10  NC06 

low  variability  high  variability 

of  size  of  size 


•Degree  to  which  the  calcifications  can  be  characterized  as  fine  linear 


0  1  2  3  4  5  6  7 

definitely  NONE  of  the 
elements  are  fine  linear 


8  9  10  NC07 

at  least  one  or  two 
elements  definitely  are,  or 
several  probably  are,  fine  linear 


•Degree  to  which  the  calcifications  can  be  characterized  as  branching 

012345678910  NCOS 

definitely  NONE  of  the  at  least  one  or  two 

elements  are  branching  elements  definitely  are,  or 

several  probably  are,  branching 


•Degree  to  which  the  calcifications  can  be  characterized  as  pleomorphic  (heterogeneous) 


0  1  2  3  4  5 

definitely  NONE  of  the 
elements  are  pleomorphic  (heterogeneous) 


6  7  8  9  10  NC09 

at  least  one  or  two 

elements  definitely  are,  or  several  probably  are, 
pleomorphic  (heterogeneous) 


•Degree  to  which  the  calcifications  can  be  characterized  as  amorphous  (indistinct) 


0  1  2  3  4  5  6  7 

definitely  NONE  of  the 
elements  are  amorphous  (indistinct) 


8  9  10  NCIO 

at  least  one  or  two 

elements  definitely  are,  or  several  probably  are, 
amorphous  (indistinct) 
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Calcifications  (Not-Definitelv-Benign^(NC^  -  cnnt. 

Elemental  Characteristics  (continued) 


•Degree  to  which  the  calcifications  can  be  characterized  as  punctate 


0  1  23456789  10  NCll 

definitely  NONE  of  the  at  least  one  or  two 

elements  are  punctate  elements  definitely  are,  or  several  probably  are, 

punctate 


•Degree  to  which  the  calcifications  can  be  characterized  as  round 
0123456789  10  NC12 

definitely  NONE  of  the  at  least  one  or  two 

elements  are  round  elements  definitely  are,  or  several  probably  are, 

round 


Distributional  Characteristics  (If  focal  or  multiple  similar  focal  distributions) 


•Number  of  elements 


less  than  5  5  to  10  more  than  10  NCI 3 

•Size  of  the  focal  distribution 

Largest  dimension  in  CC  view  _ mm  NCI 4 

Largest  dimension  in  lateral  view  _ mm  NC 1 5 


•Degree  to  which  the  distribution  can  be  characterized  as  clustered 


0  1 

2 

3 

4 

5 

6 

7 

8 

9 

10 

NC16 

definitely  NOT 

definitely 

clustered 

clustered 

•Degree  to  which  the  distribution  can  be  characterized  as  linear 


0  1 

2 

3 

4 

5 

6 

7 

8 

9 

10 

NC17 

definitely  NOT 

definitely 

linear 

linear 
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Calcifications  (Not-Definitelv-Benign^(NC^  -  cont. 


Distributional  Characteristics  (If  focal  or  multiple  similar  focal  distributions) 
(continued) 


•Degree  to  which  the  distribution  can  be  characterized  as  segmental 


0  1 

2 

3 

4 

5 

6 

7 

8 

9  10 

NCI  8 

definitely  NOT 

definitely 

segmental 

segmental 

Relationship  to  Other  Aspects  of  This  Study  (If  focal,  multiple  similar  focal, 

regional  distribution) 


•Confidence  regarding  presence  of  related  architectural  distortion 


0  1  2  3  4  5  6  7  8  9  10  NC19 

definitely  NOT  definitely 

present  present 


•Confidence  regarding  presence  of  related  mass  or  asymmetric  breast  tissue 


0  1  2  3  4  5  6  7  8  9  10  NC20 

definitely  NOT  definitely 

present  present 


Relationship  to  Prior  Study 


•This  not-definitely-benign  calcifications  finding  is:  NC2 1 

NC22 

O  new 

O  significantly  changed 
O  unchanged  or  not  significantly  changed 
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Reader  No._ 
Case  No.. 


Finding  No.. 

Skin  Calcifications  (SC) 


•Confidence  regarding  the  presence  of  skin  calcifications 


0  1  2  3  4  5  6  7 

definitely  NONE 
present 


8  9 


10 

definitely 

present 


SCOl 


•Locus  -  breast 


O  left  breast  O  right  breast 


SC02 


•Locus  -  quadrant  (check  as  many  as  apply) 

superior  medial 
superior  lateral 
inferior  medial 

^  inferior  lateral  SC03 


Relationship  to  Prior  Study 

•This  skin  calcifications  finding  is:  SC04 

SC05 

O  new 

O  significantly  changed 
O  unchanged  or  not  significantly  changed 
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Vascular  Calcifications  (VC^ 


Reader  No._ 
Case  No.. 
Finding  No._ 


•Confidence  regarding  the  presence  of  vascular  calcifications 


0  1 
definitely  NOT 
present 


2  3  4  5  6  7  8 


9  10 

definitely 

present 


VCOl 


•Locus  -  breast 


O  left  breast 


O  right  breast 


VC02 


•Locus  -  within  breast 

O  O’clock  position 


O  anterior 
O  middle 
O  posterior 


VC03A 

VC03B 


O  central  O  anterior  (subareolar) 
O  middle 
O  posterior 


O  axillary  tail 

Relationship  to  Prior  Study 

•This  vascular  calcifications  finding  is:  VC()4 

VC05 

O  new 

O  significantly  changed 
O  unchanged  or  not  significantly  changed 
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Reader  No._ 

Case  No.. 

Finding  No.. 

Coarse  (“Popcorn-Like”)  Calcifications  (CO 

•Confidence  regarding  the  presence  of  coarse  (“popcom-like”)  calcifications 
0123456789 

definitely  NOT 
present 

•Distribution  of  calcifications: 

o  single  focal  distribution  (clustered,  linear,  segmental) 

O  multiple  similar  focal  distributions 
O  regional  distribution 

O  diffuse  CC02 

•If  single  focal  distribution  or  multiple  similar  focal  distributions, 
the  shape  of  the  distribution(s)  is/are  best  described  as: 

O  clustered 

O  linear 

O  segmental  CC03 

•Locus  -  breast 

O  left  breast  O  right  breast  CC04 


10  CCOl 

definitely 

present 
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Coarse  or  “Popcorn-Like”  Calcifications  (CC^  -  cont. 


•  Locus  -  within  breast  (If  focal  or  regional  distribution) 


If  focal 

O  O’clock  position  _ 

O  anterior 
O  middle 
O  posterior 

O  central 

O  anterior  (subareolar) 

O  middle 
O  posterior 
O  axillary  tail 


CC05A 

CC05B 

If  regional  CC05C 

(check  as  many  as  apply)  CC05D 

^  superior  medial 
n  superior  lateral 
^  inferior  medial 
inferior  lateral 
central 

anterior  (subareolar) 
middle 
^  posterior 
axillary  tail 


Relationship  to  Prior  Study 

•This  coarse  (“popcom-like”)  calcifications  finding  is:  CC06 

CC07 

O  new 

O  significantly  changed 
O  unchanged  or  not  significantly  changed 
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Large  Rod-Like  Calcifications  (RL1 


Reader  No._ 
Case  No., 
Finding  No._ 


•Confidence  regarding  the  presence  of  large  rod-like  calcifications 


0  1 

definitely  NOT 
present 


2  3  4  5  6  7 


8  9 


10 

definitely 

present 


•Distribution  of  calcifications  (choose  one): 

O  single  focal  distribution  (clustered,  linear,  segmental) 
O  multiple  similar  focal  distributions 
O  regional  distribution 

O  diffuse 


RLOl 


RL02 


•If  single  focal  distribution  or  multiple  similar  focal  distributions, 
the  shape  of  the  distribution(s)  is/are  best  described  as: 

O  clustered 

O  linear 

O  segmental  RL03 

•  Locus  -  breast 

O  left  breast  O  right  breast  RL04 
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Large  Rod-Like  Calcifications  (RL)  -  cent. 


•  Locus  -  within  breast  (If  single  focal  or  regional  distribution): 


If  single  focal 


If  regional 

(check  as  many  as  apply) 


RL05A 

RL05B 

RL05C 

RL05D 


O  O’clock  position _ 

O  anterior 
O  middle 
O  posterior 

O  central 

O  anterior  (subareolar) 

O  middle 
O  posterior 
O  axillary  tail 


n  superior  medial 
superior  lateral 
inferior  medial 
inferior  lateral 
^  central 

^  anterior  (subareolar) 
^  middle 
posterior 

Q  axillary  tail 


•Number  of  elements 


less  than  5 


5  to  10 


more  than  10 


RL06 


Relationship  to  Prior  Study 

•This  large  rod-like  calcifications  finding  is:  RL07 

RL08 

O  new 

O  significantly  changed 
O  unchanged  or  not  significantly  changed 
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Reader  No._ 
Case  No.. 
Finding  No,_ 

Spherical  Lucent-Centered  Calcifications  (LC^ 


•Confidence  regarding  the  presence  of  spherical  lucent-centered  calcifications 


0  1 
definitely  NOT 
present 


2  3  4  5  6  7 


8  9  10 

definitely 

present 


LCOl 


•Distribution  of  calcifications  (choose  one): 

o  single  focal  distribution  (clustered,  linear,  segmental) 

O  multiple  similar  focal  distributions 
O  regional  distribution 

O  diffuse  LC02 

•If  single  focal  distribution  or  multiple  similar  focal  distributions, 
the  shape  of  the  distribution(s)  is/are  best  described  as: 

O  clustered 

O  linear 

O  segmental  LC03 


•Locus  -  breast 

O  left  breast  O  right  breast 


LC04 
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Spherical  Lucent- Centered  Calcifications  (LC^  -  cont. 


•  Locus  -  within  breast  (If  single  focal  or  regional  distribution) 


If  single  focal 

O  O’clock  position _ 

O  anterior 
O  middle 
O  posterior 

O  central 

O  anterior  (subareolar) 

O  middle 
O  posterior 
O  axillary  tail 


LC05A 

LC05B 

If  regional  LC05C 

(check  as  many  as  apply)  LC05D 

D  superior  medial 
Q  superior  lateral 
Q  inferior  medial 
Q  inferior  lateral 
D  central 

anterior  (subareolar) 

^  middle 
posterior 

D  axillary  tail 


•Size  of  the  spherical  lucent-centered  calcifications 


Largest  dimension  in  CC  view  _ 

mm 

LC06 

Largest  dimension  in  lateral  view 

mm 

LC07 

•Number  of  elements 


less  than  5 


5  to  10 


more  than  10 


LC08 


Relationship  to  Prior  Study 

•This  spherical  lucent-centered  calcifications  finding  is:  LC09 

LCIO 

O  new 

O  significantly  changed 
O  unchanged  or  not  significantly  changed 
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Eggshell  (Rim)  Calcifications  (EC^ 


Reader  No._ 
Case  No.. 
Finding  No._ 


•Confidence  regarding  the  presence  of  eggshell  rim  calcifications 


0  1 
definitely  NOT 
present 


2  3  4  5  6  7 


8  9 


10 

definitely 

present 


•Distribution  of  calcifications  (choose  one): 

O  single  focal  distribution  (clustered,  linear,  segmental) 
O  multiple  similar  focal  distributions 
O  regional  distribution 

O  diffuse 


ECOl 


EC02 


•If  single  focal  distribution  or  multiple  similar  focal  distributions, 
the  shape  of  the  distribution(s)  is/are  best  described  as: 

O  clustered 

O  linear 

O  segmental  EC03 


•Locus  -  breast 

O  left  breast  O  right  breast 


EC04 
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Eegshell  (Rim)  Calcifications  (EC)  -  cont. 


•  Locus  -  within  breast  (If  focal  or  regional  distribution) 


If  focal 

O  O’clock  position  _ 

O  anterior 
O  middle 
O  posterior 

O  central 

O  anterior  (subareolar) 

O  middle 
O  posterior 
O  axillary  tail 


EC05A 

EC05B 

If  regional  EC05C 

(check  as  many  as  apply)  EC05D 

superior  medial 
superior  lateral 
^  inferior  medial 
^  inferior  lateral 
0  central 

^  anterior  (subareolar) 
middle 
posterior 

n  axillary  tail 


•Number  of  elements 

less  than  5  5  to  10  more  than  10  EC06 


Relationship  to  Prior  Study 

•This  eggshell  (rim)  calcifications  finding  is:  EC07 

EC08 

O  new 

O  significantly  changed 
O  unchanged  or  not  significantly  changed 
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Reader  No._ 
Case  No.. 


Milk  of  Calcium  (MCI 

Finding  No. _ 

•Confidence  regarding  the  presence  of  milk  of  calcium 

012345678 

9  10 

MCOl 

definitely  NOT 

definitely 

present 

present 

•Distribution  of  calcifications  (choose  one): 

MC02 

o  single  focal  distribution  (clustered,  linear,  segmental) 

O  multiple  similar  focal  distributions 
O  regional  distribution 

O  diffuse  MC02 

•If  single  focal  distribution  or  multiple  similar  focal  distributions, 
the  shape  of  the  distribution(s)  is/are  best  described  as: 

O  clustered 

O  linear 

O  segmental  MC03 

•  Locus  -  breast 

O  left  breast  O  right  breast  MC04 
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Milk  of  Calcium  (MCt  -  cont. 


•  Locus  -  within  breast  (If  focal  or  regional  distribution); 


If  focal 

O  O’clock  position  _ 

O  anterior 
O  middle 
O  posterior 

O  central 

O  anterior  (subareolar) 

O  middle 
O  posterior 
O  axillary  tail 


MC05A 

MC05B 

If  regional  MC05C 

(check  as  many  as  apply)  MC05D 

superior  medial 
0  superior  lateral 
^  inferior  medial 
^  inferior  lateral 
Q  central 

D  anterior  (subareolar) 

^  middle 
^  posterior 
D  axillary  tail 


•Number  of  elements 

less  than  5  5  to  10  more  than  10  MC06 

Relationship  to  Prior  Study 

•This  milk  of  calcium  finding  is:  MC07 

MC08 

O  new 

O  significantly  changed 
O  unchanged  or  not  significantly  changed 


52 


Reader  No._ 
Case  No.. 
Finding  No._ 

Suture  Calcifications  flJCt 

•Confidence  regarding  the  presence  of  suture  calcifications 


0  12  3 

definitely  NOT 
present 

4 

5 

6 

7 

8 

9 

10 

definitely 

present 

UCOl 

•Locus  -  breast 

UC02 

O  left  breast  O  right  breast 


•Locus  -  quadrant  (check  as  many  as  apply) 

superior  medial 
D  superior  lateral 
Q  inferior  medial 

^  inferior  lateral  UC03 

Relationship  to  Prior  Study 

•This  suture  calcifications  finding  is:  UC04 

UC05 

O  new 

O  significantly  changed 
O  unchanged  or  not  significantly  changed 
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Dystrophic  Calcifications  (DC) 


Reader  No._ 
Case  No., 
Finding  No._ 


•Confidence  regarding  the  presence  of  dystrophic  calcifications 


0  1 
definitely  NOT 
present 


2  3  4  5  6  7 


8 


9  10 

definitely 

present 


DCOl 


•Distribution  of  calcifications: 

O  single  focal  distribution  (clustered,  linear,  segmental) 

O  multiple  similar  focal  distributions 
O  regional  distribution 

O  diffuse 

•If  single  focal  distribution  or  multiple  similar  focal  distributions, 
the  shape  of  the  distribution(s)  is/are  best  described  as: 

O  clustered 

O  linear 

O  segmental 


DC02 


DC03 


•Locus  -  breast 

O  left  breast  O  fight  breast 


DC04 
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•  Locus  -  within  breast  (If  focal  or  regional  distribution): 


If  focal 

O  O’clock  position  _ 

O  anterior 
O  middle 
O  posterior 

O  central 

O  anterior  (subareolar) 

O  middle 
O  posterior 
O  axillary  tail 

•Number  of  elements 

less  than  5  5  to  10 


DC05A 

If  regional  DC05B 

(check  as  many  as  apply)  DC05C 

DC05D 

n  superior  medial 
D  superior  lateral 
n  inferior  medial 
^  inferior  lateral 
Q  central 

n  anterior  (subareolar) 

D  middle 
posterior 

Q  axillary  tail 


more  than  10  DC06 


Relationship  to  Prior  StudA 


•This  dystrophic  calcifications  finding  is: 

O  new 

O  significantly  changed 
O  unchanged  or  not  significantly  changed 


DC07 

DC08 
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Reader  No.^ 
Case  No.. 
Finding  No._ 

Skin  Thickening 


•Confidence  regarding  presence  of  skin  thickening 


0  1  2  3  4  5  6  7  8  9  10  STOl 

definitely  NONE  definitely  some 

present  present 


•Locus  -  breast 


O  left  breast 


O  right  breast 


ST02 


•Visible  in  both  views  or  only  one? 

O  both 

O  one:  Oblique 

O  one:  CC  ST03 


•Locus  -  quadrant  (check  as  many  as  apply) 


If  both  views: 

0  superior  medial 

0  superior  lateral 
n  inferior  medial 

Q  inferior  lateral 

ST04A 

If  Oblique: 

^  superior 
^  inferior 

ST04B 

If  CC: 

0  medial 
n  lateral 

ST04C 

Relationship  to  Prior  Study 

•  This  skin  thickening  finding  is:  ST05 

ST06 

O  new 

O  significantly  changed 
O  unchanged  or  not  significantly  changed 
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Skin  Retraction 


Reader  No._ 
Case  No.. 
Finding  No._ 


•Confidence  regarding  the  presence  of  skin  retraction 


0  1  2  3  4  5  6  7  8  9  10  SROl 

definitely  NONE  definitely  some 

present  present 


•Locus  -  breast 

O  left  breast  O  right  breast  SR02 

•Visible  in  both  views  or  only  one? 

O  both 

O  one:  Oblique 

O  one:  CC  SR03 


•Locus  -  quadrant  (check  as  many  as  apply) 


If  both  views: 

Q  superior  medial 
superior  lateral 
inferior  medial 
^  inferior  lateral 

SR04A 

If  Oblique: 

^  superior 

0  inferior 

SR04B 

If  CC: 

D  medial 

Q  lateral 

SR04C 

Relationship  to  Prior  Study 

•This  skin  retraction  calcifications  finding  is:  SR05 

SR06 

O  new 

O  significantly  changed 
O  unchanged  or  not  significantly  changed 
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Nipple  Retraction  (NR) 


Case  No.. 
Finding  No._ 


•  Confidence  regarding  the  presence  of  nipple  retraction 


0123456789  10  NROl 

definitely  NONE  definitely  some 

present  present 


•Locus  -  breast 

O  left  breast  O  right  breast 


NR02 


Relationship  to  Prior  Study 


•This  nipple  retraction  finding  is:  NR03 

NR04 

O  new 

O  significantly  changed 
O  unchanged  or  not  significantly  changed 
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Reader  No. 


•This  axillary  adenopathy  finding  is:  AA03 

AA04 

O  new 

O  significantly  changed 
O  unchanged  or  not  significantly  changed 
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Reader  No._ 
Case  No.. 
Finding  No._ 

Trabecular  Thickening  (TT^ 

•Confidence  regarding  presence  of  trabecular  thickening 


0  1 

2 

3 

4 

5 

6 

7 

8 

9  10 

TTOl 

definitely  NONE 

definitely  some 

present 

present 

•  Locus  -  breast 

O  left  breast  O  right  breast  TT02 


Relationship  to  Prior  Study 


•This  trabecular  thickening  finding  is: 

O  new 

O  significantly  changed 
O  unchanged  or  not  significantly  changed 


TT03 

TT04 
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Reader  No._ 
Case  No.. 
Finding  No._ 

ACR  CATEGORY 

Please  select  the  appropriate  ACR  category; 

O  I  ( negative  exam,  routine  follow-up ) 

O  n  ( benign  finding,  routine  follow-up ) 

O  ni  ( probably  benign  finding,  accelerated  follow-up ) 

O  rv  (  suspicious  finding,  biopsy  recommended ) 

O  V  ( finding  highly  suggestive  of  malignancy,  biopsy  recommended ) 


OVERALL  DIAGNOSTIC  JUDGMENT 

Benign  vs.  Malignant— Rate  the  likelihood  (as  the  number  of  chances 
in  100)  that  the  finding  is  indicative  of  malignancy: 

Rating  (0  to  100)  _ 

where:  0  =  certainly  benign  or  normal 
100  =  certainly  malignant 
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APPENDIX  B  Mammography  Report 

Grammar 


The  following  is  a  portion  of  the  grammar  that  is  used  to  select  and  organize  information  in 
the  reports.  Non-terminals  to  the  left  of  the  arrow  are  expanded  to  what  is  on  the  right  of  the 
arrow.  When  there  are  choices,  those  choices  are  marked  with  conditions,  which  use  the 
numerical  values  in  the  questionnaire  to  select  the  correct  option. 

Top  Level  Organization 

REPORT  =>  COMPOSITION  FINDINGS  ASSESSMENT 
Report  Body 

COMPOSITION  SI  xondition  (OV02 0-1)  I 

52  :  condition  (OV02  2-4)  I 

53  :  condition  (OV02  5-8)  I 

54  :  condition  (OV02  9-10) 

FDJDINGS  =>  FINDING  I  FINDING  FINDINGS  I  S5 

FINDING  =>  MASS  I  ;conditioned  on  OV03 

CALCMCATION-NB  I 
CALCMCATION-B  I 
ARCHITECTURAL-DISTORTION 
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Mass 


MASS  =>  MASS-DESCRIPTION  ASSOCIATED-FINDINGS 

MASS-DESCRIPTION  =>  “There  is  a”  SIZE  TYPE+MODIFIER  “mass”  LOCATION. 

SIZE  =>  VALUE  condition  (MM06)  “cm” 

TYPE+MODIFIER  “circumscribed”  I  “lobular”  I  “glandular”  ;conditioned  on  values 

Calcifications  (Not  definitely  benign) 

CALCIFICATION-NB  =>  CALC-NB-DESCRIPTION 

Location  Specialist 

LOCATION  =>  “at”  O’CLOCK  condition  (MM03  xxx)  “in  the”  LOCUS  “breast”  I 

“in  the  “  CENTRAL  xondition  (MM03  xxx)  “position  of  the”  LOCUS  “breast”  I 
“in  the”  AXILLARY  xondition  (MM03  xxx)  “in  the”  LOCUS  “breast” 

O’CLOCK  =>  VALUE  xondition  (MM03  xxx)  “o’clock” 

CENTRAL  =>  “anterior”  xondition  (MM03  xxx)  I 
“middle”  xondition  (MM03  xxx)  I 
“posterior”  xondition  (MM03  xxx) 

AXILLARY  “axillary  tail” 

LOCUS  =»  “right”  xondition  (MMOl  R)  I 
“left”  xondition  (MMOl  L)  I 

Overall  Assessment 

ASSESSMENT  =>  IMAGE-QUALITY  REL-TO-PRIOR 

Tissue  Densitiy 

S 1  =>  “The  breast  is  almost  entirely  fat.” 
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S2  =>  “There  are  scattered  fibroglandular  densities.” 


53  =>  “The  breast  is  heterogeneously  dense.” 

54  =>  ‘The  breast  is  extremely  dense.” 

No  Findings 

55  =»  “No  masses,  significant  calcifications,  or  other  abnormalities  are  visible.” 
Relationship  to  Prior  Studies 

56  =>  “This  is  a  new  finding.” 

57  =>  “The  finding  is  significantly  changed.” 

58  ^  “The  finding  is  unchanged.” 
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Questionnaire  Analysis 

The  following  table  shows  each  question  in  the  total  questionnaire  in  the  rows  and  the  different 
specific  subparts  of  the  questionnaire  in  the  columns.  The  item  number  is  filled  in  when  a  question 
is  asked  in  that  subpart.  This  allows  us  to  see  patterns  and  isolate  specialists  in  the  text  generator 
when  the  same  information  is  elicited  in  many  different  parts  of  the  form. 


QUESTION 

MM 

AT 

TD 

AD 

NC 

SC 

VC 

oc 

RL 

LL 

EC 

ST 

SR 

RD 

AA 

TT 

confidence  regarding 
presence 

01 

01 

01 

01 

01 

01 

01 

01 

01 

01 

01 

"oT" 

"oP 

01 

__ 

01 

Locus-  breast 

02 

03 

02 

02 

03 

02 

02 

04 

04 

^04" 

04 

02 

“02" 

02 

02 

Locus  within  breast 

03 

04 

03 

03 

04 

03 

05 

05 

05 

05 

Locus-quadrant 

03 

03 

03 

Distribution  of  calc. 

02 

02 

02 

02 

02 

Distr  of  calcs 

03 

03 

03 

"03” 

Distributional:  num.  elts 

13 

06 

07 

06 

Density  wR\rt  surrounding 

03 

Confidence  about  fat 

04 

Size 

6-9 

5-6 

05 

shape 

10 

Confidence  spiculaetd 

12 

degree  of  microlobulatoin 

13 

Confidence  skin  lesion 

14 

Confidence  intrammary 

15 

Confidence  related  distortion 

16 

09 

Condifence  worrisome  calcs 

17 

07 

05 

09 

Confidence  benign  calc 

18 

08 

06 

Relationship  to  prior  study 

19 

10 

04 

09 

21 

04 

04 

06 

07 

08 

07 

04 

“oT" 

03 

”03“ 

03 

Distribution  of  breast  tissue 

02 

Conf  related  to  surgery 

Presence  related  mass 

07 

Pr.  rleates  assynmetric  tissue 

"”08“ 

20 

Variability  of  size 

Degree  fine  linear 

07 

Degree  branching 

08 

Degree  pleomorphic 

09 

Degree  amorphous 

10 

Degree  punctate 

11 

Degree  round 

12 

size  of  focal  collec 

14-15 

5-6 

Degree  collection  clustered 

16 

Degree  collecton  linear 

17 

Degree  collecton  segmental 

18 
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